Benchmarked Gemma 4 31B ARA-heretic on GMKtec EVO-X2 (Ryzen AI Max+ 395 / Radeon 8060S) using llama.cpp ROCm. While short prompts hit 8.4 t/s, 62K context runs collapsed into <unused49> and leaked across slots. Here is how upgrading the build and configuring --cache-ram resolved it.
Benchmarking five local LLM inference engines on an M1 Max 64GB with Qwen3.5-9B. Comparing GGUF and MLX decode throughput, speculative decoding gains (MTP vs DFlash), and concurrent request scaling.
We evaluated MiniCPM-V 4.7 35B-A3B on a 16GB M4 Mac mini across coding, reasoning, and table OCR benchmarks, finding 100% transcription accuracy alongside arithmetic hallucination patterns.
Tested on EVO-X2 (gfx1151) under Windows + ROCm: the official b10666 binary runs it at pp512 159.25 / tg128 23.26, and the earlier crash was my own workaround flag.
Tested on M1 Max 64GB: AtomicChat's M64 GGUF keeps the 51B N-gram table in its own shard, so a llama.cpp PR #27742 build leaves it on SSD and runs the 125B MoE at 17.6 tok/s.
Tested on GMKtec EVO-X2 (Ryzen AI Max+ 395): Q8_0 + MTP beats a 4-bit M1 Max at 22 tok/s, thinking burns 32,712 chars before any HTML, and the NSFW refusal line moves.
Tested on M1 Max 64GB: Qwen3.8-27B hits ~19 tok/s on both MLX and Ollama, but the default reasoning_effort=xhigh blew thinking up to 50,373 chars. Why Ollama dodges it.
Tested on M1 Max 64GB: hooked Qwen3.6-35B-A3B's MoE router in mlx-lm, pre-warmed the top-20 hot experts, still ~62 tok/s vs plain mmap cache. Plus the Metal OOM on Qwen3.5-122B.
Tested LFM2.5-1.2B-JP-202606 on M1 Max 64GB. llama.cpp Q4_K_M: 208 tok/s decode, JSON intact, model name hallucinated (LFM→FDM). Q8_0: 157 tok/s, no hallucination. Tool calls broken via GGUF.
35M linear projection replaces E4B's 150M 16-layer Vision Encoder. Bidirectional attention in the 48-layer LLM absorbs patch features. Comparison with Fuyu, EVE, EVEv2, and Mono-InternVL.
Hands-on with Tencent Hy-MT2 1.8B Q4_K_M (1.08GB) on M1 Max 64GB via llama-server. JSON, SRT, HTML, glossary, and minority-language prompts with full input-output pairs. The 1.25bit 440MB build does not load on stock llama.cpp 8990, and 30B-A3B (hy_v3) is not in the Mac route yet.